Real time monitoring of perceived quality of packet voice transmission

ABSTRACT

The present invention is directed to a system and method for monitoring perceived quality of a packet-switched voice service in a network. The method includes the step of receiving a packetized voice communication via the packet-switched voice service. At least one objective measurement is obtained from the received packetized voice communication. User perceived quality of voice data is derived from the at least one objective measurement. The user perceived quality of voice data is provided to a user. The steps of receiving, obtaining, deriving, and providing are performed in real-time.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation-in-part of U.S. patent application Ser. No.09/778,186 filed on Feb. 7, 2001, which is a continuation-in-part ofU.S. Pat. No. 6,370,120 the content of which is relied upon andincorporated herein by reference in its entirety, and the benefit ofpriority under 35 U.S.C. §120 is hereby claimed. The present applicationalso relates to provisional U.S. Patent Application No. 60/492,597,filed Aug. 5, 2003, which is incorporated by reference herein in itsentirety and to which the benefit of priority under 35 U.S.C. §119(e) ishereby claimed.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to telecommunications networks,and particularly to monitoring the quality of packet voice transmissionsin real-time.

2. Technical Background

New telecommunications technologies are emerging that employ packetswitching instead of the traditional circuit switched technologiesprovided by the public switched telephone networks (PSTN). These packetswitched technologies are being used to support the transmission ofdigitized voice signals over a data network such as the Internet.Providing telephone-like full duplex voice over an Internet Protocolnetwork (VoIP) is particularly important. VoIP services are attractiveto commercial long-distance carriers because they enable the use ofglobal Internet transport facilities to carry traffic that is presentlybeing carried over dedicated circuit switched facilities. The potentialbenefits of VoIP are enormous in terms of better utilization of networkbandwidth to support telephone traffic and the economies of scale fromthe use of one kind of transport for all telecommunications services.However, a major impediment to the immediate adoption of VoIP servicesrelates to the user perception of the quality of voice communicationsusing VoIP.

Traditional telephone connections have been subject to impairments inthe form of noise, attenuation, distortion, cross-talk and echo. Suchimpairments are particularly common in analog portions of the network,such as subscriber loops and frequency domain multiplexing equipment.Digital transmission alleviates many of these problems but alsointroduces quantization noise and distortions due to bit errors in thedigital signal. Even with perfect digital transmission applied to longhaul transmissions, a typical telephone connection still includes manyanalog components where impairments can occur.

A poor connection or malfunctioning piece of equipment can produceconditions that a telephone customer will find objectionable orintolerable. When there is a high incidence of poor connections,customers may complain to the service provider or to a regulatoryauthority, or simply change long distance carriers. Thus, the perceivedquality of a service provider's service is a major factor affecting thereputation and marketability of long distance telephone services.

To guard against poor quality, service providers have developed methodsto obtain objective quality measurements upon a line, piece ofequipment, or an end-to-end telephone connection. These measurements canhelp the service provider detect and gauge impairments, pinpoint weakelements, and correct deficiencies that degrade user perception ofquality. The effects of extreme fault conditions on user perception ofquality is clear. There are easily discernable thresholds for “noeffect” and “substantial degradation” conditions. As a result, theaverage consumer has to come to expect a certain quality of service fromthe PSTN.

With the proliferation of voice-over-packet technologies, maintaining aquality of service comparable to the PSTN is a major concern of serviceproviders, equipment vendors, and ultimately the consumers of packetizedtelecommunications services. Unlike circuit switched traffic, real timevoice transmission using packet switched technologies is sensitive topacket loss, packet delay, and packet jitter occurrences which arecharacteristic of packet switched networks. Packet loss and packet delayvariations may impact the ability of a voice codec to faithfullyreproduce a digitally encoded voice signal. When a received packetizedvoice transmission is missing some packets, a codec may provide an audiosignal that is distorted, garbled, or otherwise degraded.

In one approach that has been considered, the IntServ and Diffservprotocols have been proposed for improving the reliability andconsistency of packet transport. (For reference, the IntServ andDiffserv approaches are described in documents, RFC 1638 and 3317respectively, promulgated by the Internet Engineering Task Force(IETF).)

The Real Time Control Protocol (RTCP) has also been considered forobtaining real-time measurements of the receipt of packet data, and forreporting the measurements to a sender or to a network qualitymonitoring location. (For reference, RTCP is described in IETF documentRFC 1889 and in ITU Recommendation H.225.0.)

One drawback to these various approaches is that, while the reporting ofpacket arrival statistics provides some estimate of data transmissionquality, there is no consideration of the extent to which packet loss,packet delay, and packet jitter affect the perceived quality of areconstructed voice signal.

Another drawback relates to the fact that the manner in which datatransmission quality affects perceived quality of a voice channel isoften dependent on the coding scheme employed by the codec. Variouscodec schemes may exhibit differing susceptibilities topacket-transmission variations and the dependency may be quitenon-linear for some codec schemes. For example, a given packet loss rateor jitter may have very little effect on a G.711 encoded speech signal,whereas an identical packet loss rate or jitter may seriously degrade amore sophisticated G.723 encoded signal. Even if the particular codecscheme is known, an accurate model of codec behavior is required to mapthe observed packet characteristics to a perceived quality level. Theintroduction of a new standard encoding scheme, or a proprietaryencoding scheme or encryption scheme, would necessitate the developmentand deployment of new models.

For these reasons, the mere reporting of packet loss statistics isinadequate. Without accurate information about the perceived qualitylevel being experienced by users, a service provider is not certain whencorrective action is necessary or what corrective action needs to betaken to improve packet transmission performance. A service provider mayshift traffic or resources unnecessarily to improve quality of service(QOS) when, in fact, the perceived level of degradation is negligible tothe users of the service. Likewise, a seemingly minor packet deliveryproblem might be causing big problems in user perception due to theparticular codec scheme being used.

What is needed is a means for automatically monitoring and reporting innear real-time the quality of a packet voice transmission as perceivedby a user receiving voice communications.

SUMMARY

The present invention provides a system and method for automaticallymonitoring and reporting, in real-time, the perceived quality of apacket voice transmission. The present invention is versatile in that itmay be configured to report a simple raw distortion measurement orprovide sophisticated estimates of perceived voice quality, using MOSand P[UDI]. In the latter case, the present invention provides ananalytical mechanism for detecting when packet-switched voice servicesare likely to be perceived by users to be noticeably inferior tofamiliar PSTN services. This embodiment of the present invention may bebased on the VQES model described below. The present invention alsoprovides a network performance data collection and analysis system forpacket switched networks. Further, the system of the present inventionfunctions to monitor voice quality of VoIP services to support timelydetection and notification of indications of deterioration.

One aspect of the present invention includes a method for monitoringperceived quality of a packet-switched voice service in a network. Themethod includes the step of receiving a packetized voice communicationvia the packet-switched voice service. At least one objectivemeasurement is obtained from the received packetized voicecommunication. An estimate of user perceived quality of voice data isderived from the at least one objective measurement. The user perceivedquality of the received digital voice data is provided to a far end useror a service provider. The steps of receiving, obtaining, deriving, andproviding are performed in real-time. As taught herein, the objectivemeasurement may be obtained from a digital signal output from areceiving codec such that the impact of transmission impairments uponthe codec scheme are inherently taken into account in the measurement.

In another aspect, the present invention includes a system formonitoring the quality of a packet-switched voice service. The systemincludes a memory element configured to store a reference matrix thatmathematically models likely user perception of acceptable quality ofvoice service. The reference matrix is empirically derived from aplurality of objective voice measurements. A measurement probe isconfigured to obtain a plurality of test measurements for each callplaced over the packet-switched voice service. A computer is coupled tothe memory element and the measurement probe. The computer is programmedto derive a test matrix from the plurality of test measurements. Thecomputer processes the test matrix, in near real time, to determine thequality of voice over the packet-switched voice service. The quality ofservice is determined by comparing the reference matrix to the testmatrix.

In another aspect, the present invention includes a method formonitoring the quality of a packet-switched voice service in a computersystem having a graphical user interface, the graphical user interfaceincluding a display and a selection device. The method includesreceiving an alarm signal from the computer system. The alarm signal isgenerated in response to determining that the quality of voice over thepacket-switched voice service is below a predetermined level. A messageis displayed in response to the alarm signal. An amplifying display iconis selected with the selection device. A human readable description ofthe alarm signal is displayed in response to the step of selecting. Atleast one indicator of likely user perception of the quality of voicecarried over the packet-switched voice service is displayed.

In yet another aspect, the present invention includes a system formonitoring the quality of a packet-switched voice service in a network.The system includes a measurement device configured to obtain at leastone objective measurement from a packetized voice communication inreal-time. A processor is coupled to the measurement device. Theprocessor is configured to derive user perceived quality of voice datafrom the at least one objective measurement and provide the userperceived quality of voice data in real-time.

In yet another aspect, the present invention includes a method formonitoring the quality of a packet-switched voice service. The methodincludes providing a reference matrix that mathematically models likelyuser perception of acceptable quality of voice service. The referencematrix is empirically derived from a plurality of objective voicemeasurements. A plurality of test measurements are obtained for eachcall placed over the packet-switched voice service. A test matrix iscreated from the plurality of test measurements. The test matrix isprocessed, in near real time, to determine the quality of voice over thepacket-switched voice service. The quality of service is determined bycomparing the reference matrix to the test matrix.

Additional features and advantages of the invention will be set forth inthe detailed description which follows, and in part will be readilyapparent to those skilled in the art from that description or recognizedby practicing the invention as described herein, including the detaileddescription which follows, the claims, as well as the appended drawings.

It is to be understood that both the foregoing general description andthe following detailed description are merely exemplary of theinvention, and are intended to provide an overview or framework forunderstanding the nature and character of the invention as it isclaimed. The accompanying drawings are included to provide a furtherunderstanding of the invention, and are incorporated in and constitute apart of this specification. The drawings illustrate various embodimentsof the invention, and together with the description serve to explain theprinciples and operation of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a hybrid telecommunications network in accordancewith the present invention;

FIG. 2 is a diagram of an IP telecommunications network in accordancewith the present invention;

FIG. 3 is a block diagram of a system for evaluating the perceivedquality of a packet switched voice channel in accordance with anembodiment of the present invention;

FIG. 4 is a chart showing a process for providing a reference matrix inaccordance with the present invention;

FIG. 5 is a diagram illustrating the semantic content of a test matrix,a reference matrix and a reference pattern matrix in accordance with anembodiment of the present invention; and

FIG. 6 is a block diagram of a system for evaluating the perceivedquality of a packet switched voice channel in accordance with anembodiment of the present invention.

DETAILED DESCRIPTION

Reference will now be made in detail to the present exemplaryembodiments of the invention, examples of which are illustrated in theaccompanying drawings. Wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.An embodiment of the network of the present invention is shown in FIG.1, and is designated generally throughout by reference numeral 10.

The present invention is directed to a method for monitoring perceivedquality of a packet-switched voice service in a network. The methodincludes the step of receiving a packetized voice communication via thepacket-switched voice service. At least one objective measurement isobtained from the received packetized voice communication. Userperceived quality of voice data is derived from the at least oneobjective measurement. The user perceived quality of voice data isprovided to a user. The steps of receiving, obtaining, deriving, andproviding are performed in real-time. Thus, the present inventionprovides a system and method for detecting when packet-switched voiceservices are likely to be perceived by users to be noticeably inferiorto familiar PSTN services.

In an approach developed by one of the present inventors, a VoiceQuality Evaluation System (VQES) has been used to evaluate the qualityof network services. The VQES is a theoretical model used to estimateindicators of likely user perception of quality of voice carried over atelephone connection, from measurements describing the quality of thevoice connection. The inputs to the VQES are measurements of, or nominalvalues for, seven measures of a telephone connection: received speechsignal power, or test signal power attenuation; C-message noise; echopath loss and echo path delay; characteristics of natural speechwaveforms; dropped packet rates; and round trip signal transport delays.All of these objective characteristics are measured in a way that servesas a reliable indicator of when users will complain of distortion, lowvolume, echo, noise, or a noticeable delay in the flow of conversation.The resultant data sets are transformed into measures that can bereadily compared to determine whether users are likely to perceive adifference in voice quality among competing services tested. Measures ofthis kind produced by the VQES include, for example, estimates of MOSand P[UDI]. The MOS is a Mean Opinion Score (MOS) that is derived fromsubjective user evaluation of voice quality under the conditionsreflected in the distribution of the objective measurements describedabove. P[UDI] refers to the proportion of calls in the sample for whichthe subjective user evaluation of the effects of the impairments wouldbe that they rendered the call “unusable”, “difficult”, or “irritating.”As embodied herein, and depicted in FIG. 1, an example of a hybridtelecommunications network in accordance with the present invention isdisclosed. Hybrid telecommunications network 10 includes public switchedtelephone network (PSTN) 14 coupled to packet switched network 18 by wayof gateway 16. PSTN 22 is also coupled to packet switched network 18 byway of gateway 22. In the example shown in FIG. 1, standard POTStelephone 12 initiates a call with standard POTS telephone 24 by way ofcircuit switched PSTN 14. PSTN 14 routes the call to gateway 16. Gateway16 converts time division multiplexed (TDM) signals into packets thatare compatible with packet switched network 18. Gateway 16 has access toa directory of IP addresses of exit gateways. Gateway 16 uses thedirectory to select gateway 20. Packet switched network 18 transportsthe packets to gateway 20. Gateway 20 converts the packets back into aTDM format compatible with PSTN 22. PSTN 22 routes the TDM signal totelephone 24.

Packet switched network 18 is commonly referred to as a convergednetwork. A converged network is adapted to carry various types of media,such as voice, data, and audio or video streams. Each of these mediatypes are digitized and formatted into discrete packets of data. Packetswitched network 18 may be of any suitable type, but there is shown byway of example an Internet Protocol (IP) network adapted to carrypacket-switched voice, or Voice-over-IP (VoIP). In one embodiment,network 18 is the public Internet. The Internet is the term used todescribe the worldwide collection of interconnected networks that arelinked together by the use of the Internet Protocol (IP). This worldwidecollection of networks includes a host of networks sponsored byacademic, commercial, governmental, and military entities. In anotherembodiment, packet switched network 18 may include a packet switchednetwork, or a collection of packet switched networks interconnected by acommon protocol, not connected to the Internet. An example of such anetwork would be a privately owned intranet, or enterprise network. Inyet another embodiment, network 18 may include any or all of enterprisenetworks, intranets, and the public Internet.

As embodied herein, and depicted in FIG. 2, another example of atelecommunications network in accordance with the present invention isdisclosed. In this example, network 10′ is an end-to-end packet switchednetwork. IP telephone 12′ generates packet switched voice signals thatare provided directly to packet switched network 18′. The packets arerouted to IP telephone 24′ by network 18′. Those of ordinary skill inthe art will recognize that IP telephone 12′ and IP telephone 24′ mayinclude a standard telephone that is coupled to packet interfacingequipment.

Further details on hybrid and end-to-end packet-switched networks areprovided in commonly assigned patent application Ser. No. 08/751,023filed Nov. 18, 1996 and Ser. No. 08/798,350 filed Feb. 10, 1997, each ofwhich is incorporated herein by reference.

The present invention provides a real-time, or a near real-time,monitoring system for packet-switched telephone networks. The systemworks equally well with the hybrid network shown in FIG. 1 or theend-to-end packet-switched telephony system shown in FIG. 2. Those ofordinary skill in the art will recognize that both FIG. 1 and FIG. 2 aresimplifications of what a “real world” network would comprise. Muchequipment is not illustrated for the sake of simplicity and ease ofillustration. For example, the routing within the packet switchednetworks is performed by numerous routers as well as other components.

As embodied herein and depicted in FIG. 3, a system 300 for evaluatingthe quality of a packet switched voice channel is disclosed. Measurementprobe 310 is coupled to the packet-switched voice channel (not shown).Probe 310 is coupled to measurement circuitry 320. Measurement circuitryprovides computer 340 with test measurements which include objectivecharacteristics of the voice signal.

In one embodiment the objective characteristics are those reflecting ordetermining the two principal ways that packet-switching can degradeperception of voice quality. These comprise: effects of dropped packetson user perception of speech distortion, and effects of additional roundtrip delay due to packet latency on natural conversational rhythms. Theobjective measures used for illustration here are the dropped frame rateand the round-trip packet latency. These are, however, for illustrationonly, as those with ordinary skills in the art of voice qualitymeasurement will be readily able to identify alternatives that willaccomplish the same end without deviating from the present invention.

In another embodiment, probe 310 and measurement circuitry 320 areimplemented in selected routers disposed in the network. In anotherembodiment some of the computing functionality may also be disposed inthe selected router. For example, Cisco Systems, Inc. manufactures aseries of router devices commonly referred to as the Gigabit SwitchRouter 12000 Series of routers. These devices are commonly employed inthe backbone of the Internet. The 12000 Series uses 200 MHZ R5000processors. The largest model in the 12000 Series is the 12016 model,which accommodates up to 320 G/bits of data per second and can processup to 60 million data packets per second. In other embodiments, theseelements are implemented in gateways or in other network elements.Alternatively, probe 310 and circuitry 320 are implemented in astand-alone piece of equipment that is not part of the network.

The objective characteristics measured by circuitry 320 include droppedframe rate (DFR) and round trip packet-switched delay (RTL). Asdescribed earlier, each of these objective characteristics affect one ormore of a plurality of measures describing the quality of the voicesignal as perceived by users. Using techniques described below,measurements of the objective characteristics can therefore betranslated into quantifications for each of the subjectively assessedquality characteristics.

System 300 also includes a database 330 that stores a reference matrix.The reference matrix provides reference quality information for voicesignals. The reference matrix will be described in more detail below.Database 330 is coupled to computer 340. Computer 340 runs software thatestimates the likely user perception of quality information for a giveninput voice signal. Once the quality estimates are calculated, they areutilized by maintenance display 350. The functionality of display 350 isdescribed in more detail below. However, in one embodiment, display 350is coupled to real-time control circuitry. In another embodiment,display 350 provides service messages to a technician in a networkmaintenance location. Service messages could also be transmitted viapagers, e-mail, audio display and/or visual display. Those of ordinaryskill in the art will recognize that database 330, computer 340, anddisplay 350 may be co-located in a network maintenance facility 30.

Essentially, the present invention operates as follows. A referencematrix that mathematically describes a totality of conditions for whichusers are likely to find quality of a voice service acceptable is storedin database 330. As will be described in more detail below inconjunction with FIG. 4, the reference matrix is empirically derivedfrom the seven objective voice characteristics mentioned above. Afterthe test measurements are obtained by probe 310 and circuitry 320,computer 340 creates a test matrix using the measurements for aplurality of calls obtained by circuitry 320. The test matrix isprocessed, in near real time, to determine whether the quality of voiceover the packet-switched voice service for the time period of theobservations was acceptable. This determination is made by comparing thereference matrix to the test matrix. An alarm is generated if theresults of the comparison of the reference and test matrices indicatesthe possibility of unacceptable quality. The alarm notification isaccompanied by a reference pattern matrix providing an indication of thematrix cells for which the comparison was negative. Informationindicating the underlying cause and the proper response is displayed forsystem operators. A description of each of the above mentionedprocessing routines is provided below.

Referring to FIG. 4, a process for providing the reference matrix isdescribed. In particular, data flow diagram 400 shows the manner inwhich data from objective and subjective measurements of acommunications service may be processed in accordance with the presentinvention. Data flow diagram 400 initially employs non-packet switcheddata 410. Non-packet switched data 410 is derived by transformingobjective measurements 412 into perceived impairment data 416 by theapplication of transform 414. Objective measurements 412 include signalpower (PWR), C-weighted noise (NSE), echo path delay (EPD), echo pathloss (EPL), and waveform distortion (DST). Impairments 416 are analogousto subjective ratings of perceived impairments, namely low volume (LV),noise (NS), echo (EC), and speech distortion (SD). Note that variouscombinations of objective measurements 412 will affect different ones ofthe subjectively rated impairments 416. Empirical experiments undervarying conditions of signal power, noise level, and et cetera enable atransform 414 to be constructed which accurately predicts impairments416 that will reported in response to a given combination of objectivemeasurements 412. A manner in which transform 414 may be established byempirical experiments is described in U.S. patent application Ser. No.09/778,186, which is a continuation-in-part of U.S. Pat. No. 6,370,120.Both U.S. Pat. No. 6,370,120 and U.S. patent application Ser. No.09/778,186 are incorporated herein, by reference, in their entireties.

Transform 414 maps objective measurements 412 to impairments 416regardless of the actual mechanism of signal transport chosen. Anycommunications service exhibiting a particular combination of signal andnoise levels and other characteristics will be similarly rated bylisteners regardless of whether the signal is actually traversing radiosignals, PCM codecs, long-haul TDM equipment, or a packet-switched datanetwork.

Where the quality of a particular communications service is to bemeasured, many sets of objective measurements 412 may be recorded. Eachsuch set of measurements is then converted using transform 414 into aset of impairments 416. A set of impairments 416 may comprise the fourimpairments LV, NS, EC, and SD each rated by test subjects as being“none”, “some” or “much”. It is also possible to directly collectsubjective measurements of impairments 416 from a communications serviceby conducting test calls, although this practice is tedious anddisadvantageous where transform 414 is already well established. Asdescribed above, measurement set 410 includes objective measurements 412transformed into a set of impairments 416. In another embodiment,measurement set 410 includes a set of subjectively rated impairments 416directly obtained from test calls through a communications service. Alarge number of measurement sets 410 pertaining to given communicationservice may be said to comprise the service attribute test (SAT) data420 for the service. As described in U.S. Pat. No. 6,370,120, the SATdata 420 for a given communication service may be convolved with aneffects matrix 460 to yield an estimate of mean opinion score and P(UDI)for the service. Like transform 414, effects matrix 460 is a predictionmodel that may be constructed from large numbers of empiricalobservations to correlate reported impairments to overall MOS and P(UDI)values.

The relationships captured in the process displayed in FIG. 4 enableanalyses, conducted in the manner taught in U.S. Pat. No. 6,370,120, todetermine combinations of dropped frame rate (DFR) and round-trippacket-switched delay (RTL) that represent acceptable quality, relativeto extant circuit switched telephony (e.g., wherein no packet-switchingtakes place). This is accomplished as follows:

A packet loss rate (PLR) transformation 422 is applied based upon apresumed packet loss rate 424. This transformation 422 redistributes,within SAT data 420, the proportions of reporting various combinationsof impairments resulting, in this case, in the estimate of the speechdistortion impairment, augmented in relation to how much packet loss isoccurring. Transformation 422 draws upon a table 426 relating reports ofspeech distortion versus packet loss rate when all other impairments arenegligible. The result of transformation 422 is transformed SAT data430. SAT data 430 reflects the performance of a communication servicesuffering a level of packet loss rate in conjunction with whatever otherimpairments were present in SAT data 420 prior to PLR transformation422.

Transformed SAT data 430 is then convolved with effects matrix 460 inprocess block 432. The result is an estimated set of MOS and P(UDI)values 440. These values take into account the rate of packet loss 424provided as input earlier.

In step 444, the effects of packet delay are applied to the P(UDI). Theeffects of packet delay upon P(UDI) are calculated and applied to theP(UDI). It may be generally said that the greater the packet delay, themore P(UDI) is increased. It has been determined experimentally by thepresent inventors that packet delay affects P(UDI) independently ofother impairments. Therefore, it suffices to apply the effects of packetdelay after all other impairments are accounted for and to do so withoutrepeating the development of transform 414 or effects matrix 460. Afterpacket delay is taken into account in block 442, the result is thecomposite MOS and P(UDI) for the communication service with both a givenpacket delay and packet loss rate represented. The composite MOS andP[UDI] are then compared with limits corresponding to acceptable circuitswitched telephony values, to produce a profile like that shown in FIG.5. FIG. 5, of course, shows acceptable cumulative distribution functions(CDFs) for DFRs associated with the PLRs as a function of the RTL. Thismatrix is the reference matrix of the current invention.

As embodied herein and depicted in FIG. 5, a matrix 500 illustrating thesemantic content of the test matrix, the reference matrix and areference pattern matrix is disclosed. Each of these are described inmore detail below. As shown, the rows 502 are defined by variableround-trip latency intervals, in milliseconds. Columns 504 are definedin terms of variable dropped frame rate intervals, e.g., the averagenumber of frames dropped per second. Thus, the boundaries defining row502 and column 504 content can be readily changed. The intervalsrepresent lower and upper values for a range of possible values that areused to produce counts of test data elements (in the test matrix)satisfying different conditions defined by round-trip latency (RTL) anddropped frame rates (DFR) for each recorded test.

A description of the test matrix is as follows. The test matrix isconstructed by measuring a multiplicity of connections between a firstlocation and a second location. The matrix is populated with a count ofobserved calls. For example, a five (5) in element 506 indicates thatfive calls were observed having an RTL in the interval between 281-307msec, and a DFR less that 1.5.

The amount of data to be accumulated in populating the test matrices isuser specified. This is implemented via one of three control options. Inone embodiment, the number of data points to be accumulated in the testdata matrix is specified before recording the test measurements.Alternatively, the number of scheduled probe tests to be accumulated inthe test data matrix may be specified before recording. In the thirdalternative, the time period over which all available test results areto be accumulated is specified before the test data matrix is recorded.As they are created, the test results accumulated in any test datamatrix must comprise only those tests run for one particularorigin/destination pair (order considered). Data records comprising thecontent of a test result matrix shall be annotated to show: the probefrom which the tests whose results are recorded in the test data matrixoriginated; the destination probe; the date and time of the start of thefirst test whose results are accumulated in the test data matrix; andthe time of the end of the last test whose results are accumulated inthe test data matrix.

As described above, the test data matrix is processed, in near realtime, to determine the quality of voice over the packet-switched voiceservice. Interpretation of test data matrices to detect indications ofunacceptable voice quality are accomplished in the following way. Whenthe routine for processing the test data matrix, M_(ij), is invoked,each row is summed step by step, to produce a cumulative matrix S_(ij).S_(ij) is the sum of the M_(ik) values from k=1 to k=j. For example, thethird element in a particular row of cumulative matrix S_(ij) is the sumof the first, second, and third elements in the corresponding row oftest data matrix M_(ij). Likewise, the fourth element in a row ofcumulative matrix S_(ij) is the sum of the first, second, third, andfourth elements in the corresponding row of test data matrix M_(ij).This process is performed for each element in a row-by-row manner.

Each of the entries in each row of S_(ij) is divided by S_(in), which isthe last value in row S_(i), to produce a cumulative distributionfunction (CDF) matrix P_(ij). However, if the value of the last value(S_(in)) is zero, all the elements of the corresponding row in P_(ij)are populated with zeroes. Otherwise, each row in P_(ij) will comprise amonotone non-decreasing sequence of values ending with the value 1.0.

As described above, the quality of service is determined by comparingthe reference matrix to the test matrix. This step is performedindirectly. In actuality, CDF matrix P_(ij) is comparedelement-by-element to the like-sized reference matrix R_(ij), which wasdescribed in detail above. This comparison produces a reference patternmatrix, R_(ij), whose entries are defined by the following criteria:

R_(ij),=0 for all values of j in a row i, for which the values P_(ij),are all zero. In other words, when the values of P_(ij) are zero, thisindicates that no data was recorded, and the corresponding element ofRP_(ij) is set to equal zero.

When P_(ij),≠0, then RP_(ij),=1 if P_(ij),<(R_(ij)−Δ), and RP_(ij), =0if P_(ij)≧(R_(ij),−Δ).

The variable “Δ” is used to reduce statistically insignificant falsealarms produced by application of the routines that are described below.Delta “Δ” may be refined on the basis of experience. The value of Delta(Δ) is a user-specified variable or calculated function of the number oftest calls represented in the test matrix. Its purpose is to set athreshold of significance of cell-by-cell differences as might, forexample, be derived by application of the Kolmogorov/Smirnov for testsof significance of differences between CDFs.

The step of processing also includes the use of routines that areinvoked to produce associated estimates of MOS and P[UDI] from a testdata matrix. These routines are implemented by use of two auxiliaryitems. A MOS vector, VM; and a P[UDI] matrix, MP. For a given test datamatrix, M, these items shall be used to calculate estimates of MOS andP[UDI] as follows. To determine MOS, the columns of M are added toproduce the vector SC. Element-by-element multiplication of SC by VM isperformed to produce the vector VS. To determine P[UDI],element-by-element multiplication of M by MP is performed to produce thematrix SP.MOS=(ΣVS)/(ΣSC)  (1),wherein Σ denotes sum of all of the elements of an array; andP[UDI]=(ΣSP)/(ΣM)  (2).

The above combination provides valuable information because, forexample, where a MOS of 3.6 is very good and might normally beconsidered to guarantee user satisfaction, studies have shown that userswill nonetheless complain of poor quality when the P(UDI) is greaterthan six percent regardless of the MOS. Therefore, predictions ofoverall service acceptability are best made considering both MOS andP(UDI).

Referring back to FIG. 3, and in particular, to display 350, the presentinvention also includes software routines that are configured toautomatically display a notice to maintenance personnel of the detectionof a newly-generated reference pattern (RP) matrix containing at leastone non-zero value. Such a notice shall take the form of a brief alarmmessage delivered to, or generated by, a host server disposed in thenetwork maintenance facility, and written to a file of alarms. Thereference pattern matrix contains enough data to enable data recordparsing, to create a human readable amplifying display message. Thealarm message is accompanied by a display of the reference patternmatrix and/or natural language descriptors of the conditions that can beinferred from the reference pattern matrix. The human readable messagealso includes a description of the origin/destination pair associatedwith the underlying test matrix, and the date and time of the creationof the test data matrix that generated the reference pattern matrix.Receipt of the brief alarm message shall generate a visible notice ofthis information on display 350.

Maintenance facility 30 and display 350 shown in FIG. 3 include agraphical user interface (GUI). The GUI is equipped with routines thatrespond to a user selection of the alarm message described above. In oneembodiment, the graphical user interface (GUI) is equipped to show ahuman readable description of the alarm detailing the associatedorigin/destination pair, and date and time of creation of the underlyingtest data matrix. A two dimensional display of the CDF matrix P_(ij) mayalso be displayed in a format like that shown in FIG. 4. The matrixcells whose RP matrix values were non-zero, are highlighted, displayedwith a contrasting color, or otherwise visually distinguished. A line ofinformation that displays the associated values of MOS and P[UDI]generated by invoking the routines described above may also be displayedby the GUI.

Referring to database 330 in FIG. 3, the present invention also definesa compact data structure for collecting and archiving annotated testdata matrices. Computer 340, or some other maintenance facility host, isconfigured to retrieve from database archives any test data matricessatisfying user-specified criteria for origin(s) of the test data,destination(s) of the test data, or date and/or time of collection. Thedata structure for each test matrix may include origination,destination, time/date of matrix creation, and a compact representationof the matrix itself. In one embodiment, the compact representationincludes the position points of non-zero values in the matrix, and thevalues themselves.

Computer 340 is also configured to parse data retrieved from thearchived annotated test data matrices stored in database 330. Computer340 is also programmed to combine individual test data matrices toaggregate results. The aggregated results may be used by networkanalysts to detect service trends.

As embodied herein, and depicted in FIG. 6, a block diagram of system600 for evaluating the perceived quality of a packet switched voicechannel in accordance with another embodiment of the present inventionis disclosed. System 600 is configured to monitor the perceived qualityof a call between customer transceiver 200 and customer transceiver 220over packet transport network 18. Since transceiver 220 is identical totransceiver 200, only transceiver 200 will be discussed in detail. Anetwork management system (NMS) 300 is coupled to packet switchednetwork 18. Since we are mostly concerned with the receive side oftransceiver 200, the discussion will center on that aspect oftransceiver 200. Transceiver 200 (and 220) includes packet interface 202which is coupled to packet transport network 18. Packet interface 202processes the packet headers and provides codec 204 with a coded digitalsignal. Codec 204 decodes and de-compresses the coded signal andprovides D/A converter 206 with a PCM digital signal. D/A converter 206converts the digital signal into an analog electrical signal. The analogelectrical signal is converted into audible sound signal by atransducer, such as audio speaker 208.

In this embodiment, quality analyzer 610 may be coupled to packetinterface 202 and/or the output of codec 204. Quality analyzer 610 isconfigured to analyze both the 64 kbps digital signal coming from thecodec, and/or packet transmission statistical data derived from packetinterface 202. Each source may be employed to monitor the perceivedquality of the packet voice transmission.

Those of ordinary skill in the art will recognize that quality analyzer610 may be implemented in any number of ways. In one embodiment, qualityanalyzer 610 may be implemented in software resident in codec 204. Thesoftware may be executed by the processor in codec 204. In anotherembodiment, quality analyzer 610 is implemented as stand alone devicesimilar to the one depicted in FIG. 3.

In one embodiment, quality analyzer 610 is configured to analyze thedigital output of the codec to obtain an instantaneous distortionmeasurement. The distortion measurement is reported to transceiver 220via the far-end quality indicator 226 (FIG. 6). In this embodiment, thedistortion measurement need not be converted into a MOS or a P[UDI] tobe reported. The distortion may be reported as a raw measurement, as anormalized score, such as a MOS or something similar thereto, or as akurtosis value. Reference is made to U.S. Pat. No. 6,246,978, which isincorporated herein by reference as though fully set forth in itsentirety, for a more detailed explanation of obtaining distortionmeasurements directly from a digitized speech signal. In accordance withthe teachings of the '978 patent, difference values may be calculatedamong successive samples in a digitized speech signal and thedistribution of these signals may be analyzed. Abnormalities, such askurtosis, in the distribution of first and/or second differences (whichapproximate first and second derivatives) may be correlated to perceiveddistortion.

Because distortion is such an important effect, in some instances it isan effect worth reporting by itself. As explained earlier, it isconsidered to be advantageous to obtain measurements from the output ofthe codec so that the actual impact of transmission performance uponcodec function may be taken into account. This approach is alsoadvantageous because the arrangement of elements in FIG. 6 and thefunction of quality analyzer 610 need not change as new codecs aredeveloped and deployed in the future.

In another embodiment, quality analyzer 610 may be more sophisticated,having the same functionality provided by the system depicted in FIG. 3.For example, quality analyzer 610 may be configured to obtain a set ofmeasurements from the digital signal, and estimate the perceived levelof impairment using the set of measurements. The set of impairmentsinclude low volume, noise, speech distortion, and echo. Analyzer 610 maybe configured transform the impairment estimates into a Mean OpinionScore (MOS), and a probability that the calls would be considered as“unusable,” “difficult,” or “irritating” P[UDI].

As shown in FIG. 6, analyzer 610 may also obtain packet transmissioncharacteristics from packet interface 202. Packet transmissioncharacteristics include packet delay, packet loss, and packet jitter.Packet transmission characteristics may be correlated with the MOS,P[UDI], and/or the perceived impairments to determine if a perceivedquality problem is the result of a problem in packet transport network18, or in a non-packet switched portion of network 10.

Measurements of quality obtained by analyzer 610 or quality indicatorsderived by analyzer 610 may be communicated to network management system300 or to a far end transceiver 220 in the form of packetized datathrough network 18. Analyzer 610 may provide this data to packetinterface 202 to be dispatched through network 18. Far-end qualityindicator 226 may receive this data and provide a display, such as a bargraph, meter or numerical value to the far end user so that they knowimmediately how well they are being received. A far end user whoobserves a significant degradation while they are speaking may know torepeat certain portions of what they have said. Both participants mayget a sense of the quality of the connection without having to ask oneanother if they are being received clearly.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the present inventionwithout departing from the spirit and scope of the invention. Thus, itis intended that the present invention cover the modifications andvariations of this invention provided they come within the scope of theappended claims and their equivalents.

1. A method, performed by one or more devices, for monitoring perceivedquality of a packet-switched voice service in a network, comprising:receiving a plurality of packetized voice communications via thepacket-switched voice service; obtaining at least one objectivemeasurement from each of the received packetized voice communications;creating a test matrix from the objective measurements; providing an Ndimensional reference matrix that mathematically models likely userperception of acceptable quality of voice service, the reference matrixbeing derived from a plurality of objective voice measurements known toaffect user perception of voice quality, where N is greater than orequal to two; processing the test matrix to determine the quality ofvoice data over the packet-switched voice service, the quality of voicedata over the packet-switched voice service being determined bycomparing the reference matrix to the test matrix; and outputtinginformation regarding the quality of voice data over the packet-switchedvoice service, where the receiving, obtaining, processing, andoutputting are performed in substantially real-time.
 2. The method ofclaim 1, further comprising: generating at least one alarm in responseto processing the test matrix, the alarm being generated if the qualityof voice over the packet-switched voice service is below a predeterminedlevel.
 3. The method of claim 2, further comprising displaying the atleast one alarm in near real time.
 4. The method of claim 3, wheredisplaying the at least one alarm in near real time further comprises:displaying a human readable description of the at least one alarm;displaying a cumulative distribution function (CDF) matrix, the CDFmatrix being derived from the test data matrix; and displaying at leastone indicator of likely user perception of the quality of voice carriedover the packet-switched voice service.
 5. The method of claim 4, wherethe human readable description includes an origin/destination pairassociated with the test data matrix, and a date and time associatedwith the creation of the test data matrix.
 6. The method of claim 1,where providing the N dimensional reference matrix further comprises:measuring the plurality of objective voice characteristics over apacket-switched voice service to obtain sample measurements;post-processing the sample measurements to produce stable estimates ofperceived voice quality; and creating the reference matrix from thepost-processed performance data.
 7. The method of claim 1, whereprocessing the test matrix to determine the quality of voice data overthe packet-switched voice service further comprises: creating areference pattern matrix, the reference pattern matrix embodying acomparison of the test data matrix and the reference matrix.
 8. Themethod of claim 7, where processing the test matrix to determine thequality of voice data over the packet-switched voice service furthercomprises: creating a cumulative matrix, the cumulative matrix beingcreated by summing each row in the test matrix, step-by-step, such thateach element in the row is a sum of all preceding elements.
 9. Themethod of claim 8, where processing the test matrix to determine thequality of voice data over the packet-switched voice service furthercomprises: deriving a cumulative distribution function (CDF) matrixbased on the cumulative matrix, the CDF matrix being created by dividingeach element in each row of the cumulative matrix by a largest value ineach row of the cumulative matrix.
 10. The method of claim 9, whereprocessing the test matrix to determine the quality of voice data overthe packet-switched voice service further comprises: comparing eachelement of the CDF matrix with each corresponding element of thereference matrix to create the reference pattern matrix.
 11. The methodof claim 10, where an element of the reference pattern matrix ispopulated with a zero value when either a corresponding value of the CDFmatrix is zero, or if the corresponding value of the CDF matrix isgreater than a predetermined value.
 12. The method of claim 10, where anelement of the reference pattern matrix is populated with a value of one(1) if the corresponding value of the CDF matrix is less than apredetermined value.
 13. The method of claim 1, where processing thetest matrix to determine the quality of voice data over thepacket-switched voice service further comprises: computing a meanopinion score (MOS) corresponding to a subjective user evaluation of thequality of voice over the packet-switched voice service.
 14. The methodof claim 13, where processing the test matrix to determine the qualityof voice data over the packet-switched voice service further comprises:computing a percentage of calls users would find unusable, difficult, orirritating P(UDI), the P(UDI) corresponding to a subjective userevaluation of the quality of voice over the packet-switched voiceservice.
 15. The method of claim 1, further comprising: storing the testdata matrix and associated data created during the processing.
 16. Themethod of claim 1, further comprising: defining a data structure forcollecting and archiving annotated test data matrices.
 17. The method ofclaim 16, where the data structure includes an identification of thepacket-switched voice service associated with a test data matrix, datacorresponding to the test data matrix, and a time and a date the testdata matrix was created.
 18. The method of claim 1, where the objectivemeasurements include measurements of a plurality of objective voicecharacteristics.
 19. The method of claim 18, where the plurality of testmeasurements include measurements of a dropped packet rate (DPR) andround trip packet latency (RTL).
 20. The method of claim 1, whereobtaining the at least one objective measurement from each of thereceived packetized voice communications, further comprises: obtainingat least one objective measurement of a reconstituted digitalrepresentation of the received packetized voice communication, thereconstituted digital representation being obtained from a receivercodec.
 21. The method of claim 20, where the at least one objectivemeasurement includes determining differences among successive samples inthe reconstituted digital representation.
 22. The method of claim 1,where outputting the information regarding the quality of voice dataover the packet-switched voice service comprises: transmitting theinformation regarding the quality of voice data over the packet-switchedvoice service to a quality indicator disposed in a user transceiver set.23. The method of claim 1, where outputting the information regardingthe quality of voice data over the packet-switched voice servicecomprises: transmitting the information regarding the quality of voicedata over the packet-switched voice service to a network managementsystem.
 24. The method of claim 1, where outputting the informationregarding the quality of voice data over the packet-switched voiceservice comprises: providing a raw distortion measurement.
 25. Themethod of claim 1, where outputting the information regarding thequality of voice data over the packet-switched voice service comprises:providing a normalized score corresponding to a distortion measurement.26. The method of claim 21, where outputting the information regardingthe quality of voice data over the packet-switched voice servicecomprises: determining a kurtosis value of a distribution of thedifferences.
 27. A system for monitoring a quality of a packet-switchedvoice service in a network, comprising: a memory element configured tostore an N dimensional reference matrix that mathematically modelslikely user perception of acceptable quality of voice service, thereference matrix being derived from a plurality of objective voicemeasurements known to affect user perception of voice quality, where Nis greater than or equal to two; a measurement probe configured toobtain a plurality of test measurements for each call placed over thepacket-switched voice service; and a computer coupled to the memoryelement and the measurement probe, the computer being configured to:derive a test matrix from the plurality of test measurements, processthe test matrix, in near real time, to determine the quality of voiceover the packet-switched voice service, the quality of voice over thepacket-switched voice service being determined by comparing thereference matrix to the test matrix, and output information regardingthe quality of the voice over the packet-switched voice service.
 28. Thesystem of claim 27, further comprising a network maintenance systemcoupled to the computer, the network maintenance facility beingconfigured to generate at least one alarm in response to an inputreceived from the computer, the alarm being generated if the quality ofvoice over the packet-switched voice service is determined to be below apredetermined level.
 29. The system of claim 28, where the networkmaintenance system further comprises at least one display, the at leastone display being configured to display the at least one alarm in nearreal time.
 30. The system of claim 28, further comprising a graphicaluser interface including a display and a selection device, the graphicaluser interface being configured to: display a human readable descriptionof the at least one alarm in near real time; display a cumulativedistribution function (CDF) matrix, the CDF matrix being derived fromthe test data matrix; and display at least one indicator of likely userperception of the quality of voice carried over the packet-switchedvoice service.
 31. The system of claim 30, where the human readabledescription includes an origin/destination pair associated with the testdata matrix, and a date and time associated with the creation of thetest data matrix.
 32. The system of claim 27, where the computer isfurther configured to create a reference pattern matrix, the referencepattern matrix embodying a comparison of the test data matrix and thereference matrix.
 33. The system of claim 32, where the computer isfurther configured to create a cumulative matrix, the cumulative matrixbeing created by summing each row, step-by-step, such that each elementin the row is a sum of all preceding elements.
 34. The system of claim33, where the computer is further configured to derive a cumulativedistribution function (CDF) matrix from the cumulative matrix, the CDFmatrix being created by dividing each element in each row of thecumulative matrix by a largest value in each row of the cumulativematrix.
 35. The system of claim 34, where the computer is furtherconfigured to compare each element of the CDF matrix with eachcorresponding element of the reference matrix to create a referencepattern matrix.
 36. The system of claim 35, where an element of thereference pattern matrix is populated with a zero value when either acorresponding value of the CDF matrix is zero, or if the correspondingvalue of the CDF matrix is greater than a predetermined value.
 37. Thesystem of claim 35, where an element of the reference pattern matrix ispopulated with a value of one (1) if the corresponding value of the CDFmatrix is less than a predetermined value.
 38. The system of claim 27,where the computer is further configured to compute a mean opinion score(MOS) corresponding to a subjective user evaluation of the quality ofvoice over the packet-switched voice service.
 39. The system of claim27, where the computer is further configured to compute a percentage ofcalls users would find unusable, difficult, or irritating P(UDI), theP(UDI) corresponding to a subjective user evaluation of the quality ofvoice over the packet-switched voice service.
 40. The system of claim27, where the computer is further configured to store the test datamatrix and associated data created when the computer processes the testmatrix.
 41. The system of claim 27, where the computer is furtherconfigured to collect and archive annotated test data matrices in a datastructure.
 42. The system of claim 41, where the data structure includesan identification of the packet-switched voice service associated with atest data matrix, data corresponding to the test data matrix, and a timeand a date the test data matrix was created.
 43. The system of claim 27,further comprising a data base configured to store reference matrices,test measurements, and/or test matrices.
 44. The system of claim 28,where the network maintenance facility is further configured to: displaya message in response to generating the alarm; receive a selection of anamplifying display icon by a selection device; display a human readabledescription of the alarm in response to selection; and display at leastone indicator of likely user perception of the quality of voice carriedover the packet-switched voice service.
 45. The system of claim 44,where the at least one indicator includes a mean opinion score (MOS).46. The system of claim 44, where the at least one indicator includes apercentage of calls users would find unusable, difficult, or irritatingP(UDI), the P(UDI) corresponding to a subjective user evaluation of thequality of voice over the packet-switched voice service.
 47. The systemof claim 44, where the at least one indicator includes a distortionindicator.
 48. A method, performed by one or more devices, formonitoring perceived quality of a packet-switched voice service in anetwork, the method comprising: providing an N dimensional referencematrix that mathematically models likely user perception of acceptablequality of voice service, the reference matrix being derived from aplurality of objective voice measurements known to affect userperception of voice quality, where N is greater than or equal to two;obtaining a plurality of test measurements for each call placed over thepacket-switched voice service; creating a test matrix from the pluralityof test measurements; processing the test matrix, in near real time, todetermine the quality of voice data over the packet-switched voiceservice, the quality of voice data over the packet-switched voiceservice being determined by comparing the reference matrix to the testmatrix, and outputting information associated with the quality of voicedata over the packet-switched voice service.